The present invention generally relates to the field of capturing facial expressions and cognitive states of users wearing a head-mounted display (HMD).
The recent introduction of consumer-level HMDs has led to a revival in virtual reality and is drawing wide interest from consumers for gaming and online virtual worlds. With the help of existing motion capture and hand tracking technologies, users can navigate and perform actions in fully immersive virtual environments. But users lack a technological solution for face-to-face communication that conveys compelling facial expressions in virtual environments.
Facial animation has been mostly dominated by optical capture systems that use cameras or depth sensors. Methods for facial representations, tracking, mapping, and animation have been developed, greatly impacting film and game production. However, because a typical face is more than 60% occluded by a HMD, established optical sensing methods that achieve the desired facial tracking quality fail to capture nearly the entire upper face. Also, typical uses of HMD in virtual reality involve head and body rotations and large amplitude movements, making it difficult to capture facial expressions by a stationary camera or a set of stationary cameras.
There also lacks a technological solution for effective cognitive and emotional state monitoring of the user of the HMD, which can be beneficial to the user for both health purposes and enhancing the immersiveness of the virtual reality experience. For example, monitoring the user's fatigue while a user is playing a game in the virtual reality environment could prevent excessive use of the HMD that may be harmful to the user.